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ABSTRACT 

vir tual Environment technology is a display and control technology that can surround a person 
in an interactive computer-generated or computer-mediated virtual environment. It has evolved at 
NASA Ames Research Center since 1984 to serve NASA’s missions and goals. The exciting 
potential of this technology, sometimes called Virtual Reality, Artificial Reality, or Cyberspace, 
has been recognized recently by the popular media, industry, academia, and government 
organizations. Considerable research and development will be necessary to bring it to fruition. 

INTRODUCTION 

Virtual Environment technology provides powerful user interface techniques for use in NASA's mission- 
oriented applications, including computational fluid dynamics, planetary data visualization, in-space 
telepresence, and space station telerobotics. In a broader context, several years of international interest in 
NASA’s Vir tual Environment technology and related user interfaces has resulted in massive media coverage 
(e.g. References 1 -8), and has encouraged several American companies to develop related commercial 
products 9 . Using head-tracked head-mounted displays, a Virtual Environment system provides the user with 
a vivid experience of three-dimensional space. It can be used with computer graphics systems as a personal 
simulator, surrounding the user with a virtual interactive environment. Alternatively, it can utilize head- 
slaved cameras, and other sensors, to provide telepresence. In either case, an instrumented glove or similar 
device may be used to detect hand shape and position, so as to enable the user to manipulate objects in 
computer generated or remote environments. 

The human desire to create virtual environments, synthetic experiences, and artificial realities is an ancient 
dream of emperors, has a rich recent history with many talented contributors, is a central theme of 
technology, and is a fundamental aspect of human thinking.* 9 The author's original inspiration to actually 
create virtual environments derived, appropriately enough, from popular media coverage of NASA s 
unmanned Surveyor missions to the moon in the mid-1960's. In July 1966, Life magazine earned 
photographs of one of NASA's first virtual environment displays, spherical mosaicked panoramas that 
allowed scientists to "see what Surveyor's swiveling camera saw." Mimicking the technique, the effects 
were dramatic. 

Today, NASA’s Virtual Environment technology allows scientists to explore planetary environments using 
head-tracked head-mounted displays and computer generated imagery. The computer graphics technology 
that enables this was originally developed by such pioneers as Ivan Sutherland. His head-mounted display 
project 12 in the mid-1960's served not only as further inspiration, but it also led to the development of the 
foundations of computer graphics techniques and systems. While multi-million dollar military systems 
have used head-mounted displays in the years since Sutherland's work, the notion of a personal virtual 
environment system as a general purpose user-computer interface was temporarily neglected. In 1984, 
NASA was able to capitalize on emerging technologies in order to make a dramatic cost reduction in the 
essential display system, and thus to create the Virtual Environment Workstation 1 ^. The exciting potential 
of Virtual Environment technology has now inspired many people to carry it forward to greater capability. 

DESCRIPTION OF THE TECHNOLOGY 

A virtual workstation is a general purpose human -information interface device for creating virtual 
environments that can be optimized for use in specific applications. It is a generalization of today s desktop 
workstation (with its display screen, keyboard, and mouse). Such workstations provide the mechanism 
whereby humans interactively process information. The virtual workstation greatly increases the ways in 
which information can be handled, and can provide a far more intuitive means of interaction with 
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information. The entire virtual three-dimensional world of the user becomes the display and controls. 
Every object or data point in that virtual world can be an interactive device. Each can be manipulated by 
touch, by gesture, or by voice. This vastly broadens the range of expression between the user and the 
information system far beyond the mere typing of keys or the two dimensional movements of a mouse 14 . 

A virtual workstation consists of body-ported devices and the computer/video system hardware and software 
that supports and integrates them. The user wears a viewer consisting of a video display screen (with its 
electronics) and wide-angle magnifying lenses. The viewer is supported by a box of electronics for power 
supply, signal conversions, and functional adjustments and accessories. The viewer is mounted so as to 
replace the visual field of the user. (A few versions allow the imagery to be combined with the teal visual 
environment of the user.) The head-mounted unit also carries a microphone, earphones, and a head-position 
and orientation sensor. Typically, the hands are fitted with shape, position, and orientation sensors. 
Ultimately, the entire body could be tracked. 

A host computer integrates the peripheral units and coordinates their interaction. A computer graphics 
system generates three-dimensional objects for display in the viewer. The imagery is altered in accordance 
with the head-movements of the user to provide a stable visual environment Remote video cameras can be 
slaved to the motions of the user's head, providing a remote source of imagery for visual telepresence. Hand 
gestures are also interpreted as desired for alteration of the appearance of the scene, behavior of objects, or 
other interactions with the environment Telerobotic systems can be controlled by the motions of the user's 
hands, providing remote manipulation. Peripheral systems convert voice commands to computer 
commands, convert computer output to synthetic speech or other sounds, and convert body movements to 
computer input. Video recorders can record what the user sees and hears for non-interactive replay later. 
Force feedback can be used to allow remote or synthetic objects to feel more solid and real. 

The software consists of many reusable modules of computer commands, and various special purpose 
programs which use these modules and also contain considerable amounts of unique computer code. Digital 
models of various task environments, such as the Space Shuttle, Space Station, and planetary terrain, are 
created and stored in databases. They are displayed and manipulated by the software programs under control 
of the user wearing the virtual workstation. 

The virtual workstation will find widespread use in applications involving highly spatial information. 
These include planetary exploration, telerobotics, computer-aided-design in general and architecture in 
particular, and scientific visualization. It will also impact applications which can be cast into a spatial 
context (e.g. three dimensional libraries of text or other "dataspaces"). Further, it may be feasible to create 
virtual, dynamic three-dimensional worlds from inherently non-spa tial data so as to ease interpretation and 
interaction. Virtual environment systems will greatly influence art, entertainment, education, medicine, and 
many other fields. 

MISSION-ORIENTED RESEARCH UTILIZING THE VIRTUAL WORKSTATION 

The creation and generic development of the NASA Virtual Environment Workstation was accomplished 
under the base research and technology program in Space Human Factors of the NASA Office of Aerospace 
and Exploration Technology . After several years of generic development and demonstrations, the focus is 
now on refining the device and its style of interaction for specific NASA applications. 

A cooperative project has been established with the Applied Research Office of the Numerical Aerodynamic 
Simulation (NAS) Systems Division at NASA Ames to develop the Virtual Workstation for NAS 
applications. Computational fluid djmamicists will utilize the Virtual Workstation in their analyses, while 
human factors researchers will investigate improved interfaces for the specific visualizations important to 


The objective of the Visualization for Planetary Exploration program is to conduct research and 
development of crew interfaces for terrain exploration systems. The approach has been: 1) to review 
mission operational experience, mission constraints and opportunities, and the state-of-the-art in exploration 
technology , 2) to investigate user behaviors and requirements, and, 3) to enlist interdisciplinary expertise to 
develop, implement, demonstrate, and evaluate advanced crew interfaces. Field studies have been conducted 
in desert terrain of interest to planetary geologists and mission planners in order to understand operations 
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central to planetary surface exploration 1 ^. In addition, a planetary terrain visua liz ation testbed is under 
development to support focussed user interface research. This powerful compute- system provides dynamic 
interaction with planetary terrain data, currently Mars and Earth, but is easily applicable to Venus, Earth s 
moon, or other planetary bodies. 

Planetary geoscientists have expressed a strong scientific interest in the development of Virtual 
Environment technology for planetary surface exploration using telepresence 16 . A leading concept is to 
extend the reach of human exploration beyond the perimeter of manned EVA by use of telepresence from 
central manned bases to numerous unmanned rovers 1 ^. Geoscientists with decades of experience in field 
work believe that human presence contributes a tightly coupled and essential interplay of cognitive, 
perceptual, manipulative, and locomotor skills that can complement automated rover operations. By using 
telepresence, these unique human skills may be applied over a greater area at an earlier date. 

The Virtual Environment technology is of interest to the Office of Space Flight for possible use on Space 
Station Freedom as it evolves. Under the Advanced Development Program, a study is underway at 
Marquette University and Astronautics Corporation of America to determine the design accommodations, 
also known as "hooks and scars”, that may be required to support eventual use of the Virtual Environment 
technology onboard Space Station Freedom. Late phases of the study will determine the cost versus 
benefit trade-offs, and operationally realistic prototypes will be developed. 

RESEARCH AND DEVELOPMENT ISSUES 

The key research issues in Virtual Environment technology involve the human factors of body-ported 
displays and controls, the specific needs of the users in the targeted applications, and the computer sciences 
of system architecture, data management, and compute graphics. Finding the most appropriate engineering 
tradeoffs for a specific application is one of the most challenging problems. This requires a thorough 
knowledge of the visual, proprioceptive, vestibular, tactile, and auditory interfaces, including both the 
human sensory channels and the system capabilities available or required to interact with these channels. 
Cutting across this view of the problem are the needs, constraints, and opportunities of the application, the 
user, and the mission. 

The special requirements and constraints of this work center around the comprehensive view of the user as 
an intelligent actor and multi-channel information processor in a completely programmable information 
environment surrounding the user. Some have referred to this kind of interactive environment as an 
artificial reality, which certainly implies a massive computational demand. This places a significant burden 
on the hardware and software of the system. 

Most existing user interface equipment is designed for a far less ambitious kind of interaction and a far more 
constrained model of the user. Thus, prototypes must be built up from systems that were not necessarily 
intended to function together or for our purposes. In addition, the information bandwidth of the human 
operator is far in excess of current technology, particularly with respect to real-time rendering of realistic, 
interactive objects and environments. This places significant constraints on the application of virtual 
environment interfaces. The challenge is to integrate and apply the best possible system within the many 
constraints, while also working to advance the technology so as to eliminate the most restrictive 
constraints. 


Current impediments to commercial applications 

Before development of Virtual Environment (VE) based commercial applications is reasonable, several basic 
improvements in the hardware and software are required. The necessary improvements involve 
enhancements to the functionality and integration of the system components, and providing for adequate 
complexity of the interactive virtual environment. Given sufficient technical capability, the utility of VEs 
for real applications must be evaluated in detail. The impediments include limited resolution viewers, lack 
of available digital models, poor system integration and standardization, and the lack of careful attention to 
the specific applications. 

The visual resolution of available viewers is too low and/or the cost is too high. There have been no mass 
produced VE viewers, and those that are available are crude. As a result, VE systems suffer from limited 
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applicability to real-world problems. A mass produced viewer is needed that has high resolution, wide field 
of view, and low cost 


There are no readily accessible collections of interactive digital models to populate virtual environments. 
Instead, each group must generate these models from scratch. This results in ad hoc solutions, poor 
commonality, redundant effort, and limited extensibility. A comprehensive effort is needed to ensure 
effective design, generation, and distribution of digital models for VEs. 

Comprehensive and integrated hardware and software tools for creating VEs are not available. As a result, 
many potential contributors cannot yet become involved in VE research and development. Currently, the 
many components of a VE system must be pieced together from existing technology, resulting in poorly 
integrated systems, ad hoc architectures, and poor commonality. Most current networks, computer 
graphics systems, processors, and peripherals were not specifically designed for use in VEs, and the "wrong" 
tradeoffs have been sometimes been made. This results in fundamental limitations in the architecture of 
current (ad hoc) VE systems. New approaches must be explored and optimal VE architectures must be 
identified and implemented. A collective effort should promote standardization, while retaining 
extensibility and the ability to integrate proprietary tools. 

In parallel to the technical impediments outlined above, there has not yet been sufficient focussed effort by 
industry to determine the potential of VEs in commercial applications. As a result, there is limited 
recognition (outside of Japan) of the potential of this technology. American industry must investigate 
commercial applications of Virtual Environment technology. 

Research and development to advance commercial applications 

Research and development to advance commercial applications of virtual environment technology should 
concentrate on four major areas: 1) devices, 2) modelling, 3) interactivity, and 4) user requirements. 

VE devices are the hardware and software that enable the user to experience and interact with a vir tua l 
environment VE devices includes all of the physical hardware and all of the genual purpose system 
software. Software unique to an application is also included in this component, but is likely to be 
proprietary. The objectives of work on VE devices should be to create and develop integrated, modular, 
extensible hardware and software tools for building VEs; to develop tools for VEs that are capable of 
benefitting high payoff applications; and to make a common set of tools available. And it is important to 
stress the importance of supporting real-time interactions in Virtual Environments. 

Modelling is the acquisition and generation of digital models of environments and objects, along with their 
attributes and behaviors, which are the contents of a VE, as well as their archiving, distribution, 
manipulation, and modification. This includes all of the hardware and software that are specific to 
modelling. The overall objective in modelling work for VE should be to develop tools and techniques that 
provide for the effective design, capture or creation, and distribution of interactive digital models of objects 
and environments for VEs. 

Interactivity refers to the style or "look and feel” of the VE, the metaphors 18 by which information is 
organized, the behaviors of objects and the "physics" of the environment as they appear to the user, and the 
collection of ways that the user may interact with virtual objects and the virtual environment Research 
objectives to support commercial applications should include optimization of the effectiveness of the user 
in accessing, understanding, and acting upon complex information in Virtual Environments. And it is 
obviously important to support ease of use, robustness, user friendliness, and a minimal need for manuals. 
The nature of the appropriate "look and feel", metaphors, behaviors, and "physics" are all open questions, 
and depend upon the applications. 

User requirements are a set of specifications based on user needs, problems, constraints, and opportunities 
that guide the design and implementation of tools intended to help users. Work done in this area will have 
a significant impact on the design of application hardware and software, of course, but will also drive the 
interactivity paradigms. Objectives in the area of user requirements should be: identify the needs and 
problems of users that might be addressed and solved by VE technology; conduct field studies of users in 
selected candidate applications; develop application testbeds that address real user requirements; and provide 
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feedback to implementers of VE devices, especially viewers, trackers, and other devices that directly impact 
on the user's working environment 


CONCLUSION 

Virtual Environment technology has tremendous potential, but considerable research and development is 
necessary to bring it to fruition. However, the media hype might lead one to believe that the technology is 
nothing more than "goggles and gloves." This emphasis is understandable given that this hardware is the 
most obviously new physical part of virtual environment interfaces. The real revolution, once there are lots 
of goggles and gloves around, will be in the generation of, and interaction with, computer-generated digital 
worlds, and the ability to be telepresent in remote locations via computer mediation. This will influence 
NASA’s missions and goals, for example, by allowing everyone to personally explore digital models of the 
canyons of Mars, or by enabling astronauts to do routine maintenance while safely inside the space station. 
It will also greatly influence art, entertainment, education, medicine, and many other fields. The Virtual 
Environment team at NASA is proud to have contributed to this advancement in human-centered 
technology. 


TEAM MEMBERS 

The Virtual Environment Research program has been made possible by the combined efforts and dedication 
of a diverse team of researchers, each of whom has made unique and substantial contributions. The group is 
devoted to research and development of the concepts, implementation, and applications of virtual 
environment technology that serve NASA’s missions and goals. 

NASA Ames Research Center 


Dr Michel M rfirftf *- vv is the originator of "Virtual Environment" user interface research at NASA. He is a 
research scientist, and is Principal Engineer of the Human Interface Research Branch. He leads the 
Visualization for Planetary Exploration program, the Telepresence User Science Requirements project, 
and the Space Station Advanced Development program in Virtual Environment displays. Since 1981 
he has conducted research in spatial information transfer at NASA. In 1984, he created the Virtual 
Visual Environment Display (VIVED) program at Ames and designed the VIVED system with the help 
of Humphries, Eriskin, and Dear don. By early 1985, he designed and developed NASA s first Virtual 
Environment Workstation, a personal simulator and telepresence device consisting of a host computer, 
an interactive computer graphics system, video imaging technology, and the VIVED system. Between 
July 1985 and August 1987, McGreevy was temporarily assigned to NASA Headquarters in 
Washington D.C, as Program Manager for NASA's research programs in Aerospace Human Factors and 
Aeronautical Computer Science. For his research management activities in Washington, he won 
NASA's Special Achievement Award. Since the beginning of the Virtual Environment project in 
1984, he has actively promoted technology transfer within NASA, and to other government 
organizations, industry, and academia. 

Dr. Stephen R. Ellis is a research psychologist with a long history of research in spatial instruments. In 
1981, he and McGreevy began a program of research in spatial information transfer, emphasizing the 
interpretation of perspective displays. He is currently developing research projects utilizing the Virtual 
Workstation for part-task experiments, including studies in manual tracking and stereo vision. He is 
also an adjunct professor in the Optometry Department at UC Berkeley. 

Dr Beth Wenzel contributed the concept of three-dimensional sound and auditory symbology to the Virtual 
Workstation project Together with university scientists, she is investigating the science of 3D sound 
and applying her findings to the project 

Py Tim T jrimf.r joined the project in 1990 and is currently working to upgrade the viewer technology and 
plans to develop additional refinements based on parameters of human vision. 
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Research Institute for Advanced Computer Science (RIACS) 

Dr. Lew Hitchner is a computer scientist and computational geographer, and is responsible for software 
implementation of a user interface testbed to support research in visualization for planetary exploration. 
He was hired by RIACS, an institute of the Universities Space Research Association, in October 1989 
specifically to work under contract to NASA on the Visualization for Planetary Exploration program. 
Prior to this, he was a professor of computer graphics at UC Santa Cruz. Hitchner earned his Ph.D. in 
Computer Science at University of Utah. 

Sterling Federal Systems 

Amy Wu assisted McGreevy in programming NASA's first Virtual Workstation, and is currently lab 
systems manager and applications programmer for the Visualization for Planetary Exploration program. 

Jim Humphries, Design Engineer and Hardware Lab Manager, co-designed (with Saim Eriskin) the 
electronics for the original VIVED system and for the subsequent refinements of the original design, 
including the first boom-mounted system. 

JQC Peardon, Senior Technician, has assisted Jim Humphries and Saim Eriskin in implementation of 
several generations of the VIVED system since the beginning of the project. 

Rick Jacoby has been a research programmer on the program since 1988. He programmed an interactive 
telerobotics demonstration on the Virtual Workstation and is currently enhancing it for use as a research 
tool. 

Phil Stone is a programmer working for Beth Wenzel who has been with the project since 1988. 

UC Berkeley 

Romy Bauer is a graduate student in computer science. After receiving a BS in computer science and a BS 
in geography at UC Berkeley, she worked full time for a year (1989-1990) developing terrain analysis 
and visualization techniques for the VPE project. She is currently a half-time research assistant on the 
project. 


Other Key Technical Contributors 

Eric Howlett of Pop-Optix Labs designed and built the wide angle stereo optics. 

Shigeru Morokawa. R&D Manager of the Technical Research Lab, Citizen Watch Company, designed, 
custom-built, and specially modified many of the liquid crystal display systems needed for the program. 

Dr. Xin Feng. Assistant Professor, Dept, of Electrical, Computer, and Biomedical Engineering, Marquette 
University, Milwaukee, WI, is Principal University Investigator on the Space Station Vir tual 
Environment technology project. 

Richard Foster. Manager, Space Systems Dept., Astronautics Corp. of America, Madison, WI, has over 25 
years of systems engineering experience and is Principal Industry Investigator on the Space Station VE 
technology project 


Ex-Members of the Team 

Saim Eriskin (Sterling) served as Senior Engineer and Electronics Designer from the first day of the project 
in 1984 until October 1990. He co-designed (with Jim Humphries) the electronics for the original 
VIVED system and for the subsequent refinements of the original design, including the first boom- 
mounted system. 

Scott Fisher (NASA) was hired in 1985 to join the growing project team and left NASA in early 1990. He 
brought the VPL Research Dataglove to the Virtual Workstation, and provided a creative outlook and 
familiarity with innovative human interface projects and researchers. His experience includes 
association with related activities at Atari Research and MITs Media Lab. Mr. Fisher served as lab 
manager and point of contact at Ames during fiscal years 1986 and 1987. 

Jim Tanner and Dave Kaiser (Sterling) woe Project Managers during fiscal years 1986 and 1987. 

Doug Kerr (Sterling) designed software libraries and database tools, and provided expertise in systems and 
applications software which were critical to the success of the project between 1985 and 1987. 

Wairen Robinctt (NASA) contributed his extensive software skills and demonstration software 1985-1987. 
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Stephen Brvson (Sterling) was a programmer on the Virtual Workstation program between 1988 and 1990. 
He is currently working with the Numerical Aerodynamic Simulation staff on a spinoff virtual 
environment project 

David Kohlas (RIACS) was a research programmer on the VPE program during the summer of 1990. He 
developed significant application software refinements including improved interactive capabilities. 

Mark Bolas (Fake Space Labs) designed and built a motor-driven camera platform in 1989. 

Other ex-members include: Cordell Ratzlaff . Ian McDowell. Dr. Bob Brown. Ken Uil la nd , Stcff a n Je f f ST S. 
Clav Coler . A1 Duncan , and a long list of students and part-time programmers. 
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